Skip to content

[APMSVLS-469] feat(traces): rescue errored traces via agent-side error sampler - #1320

Open
lucaspimentel wants to merge 13 commits into
mainfrom
lpimentel/add-trace-error-sampler
Open

[APMSVLS-469] feat(traces): rescue errored traces via agent-side error sampler#1320
lucaspimentel wants to merge 13 commits into
mainfrom
lpimentel/add-trace-error-sampler

Conversation

@lucaspimentel

@lucaspimentel lucaspimentel commented Aug 3, 2026

Copy link
Copy Markdown
Member

TL;DR

Rescues errored traces that automatic sampling drops on the lambda_extension_compute_stats path, so errors stay visible even under aggressive sampling. Ships disabled behind DD_APM_ERROR_SAMPLER_ENABLED, using the AlwaysKeep mode of the new datadog-agent-trace-sampler crate (DataDog/serverless-components#141). Draft until that crate merges.


Overview

On the lambda_extension_compute_stats path, the extension drops every P0 (priority <= 0) trace after computing its stats, so an errored trace the tracer sampled away is invisible in the UI even though its stats are counted. This adds an agent-side error sampler that rescues those traces and stamps _dd.errors_sr on the root span, matching the Go trace agent's ScoreSampler / ErrorTPS behavior.

Candidates, matching the Go agent:

  • An error on any span (not just the root) qualifies the trace.
  • Only automatic drops are candidates — explicit drops (a sampling rule or MANUAL_DROP) are always honored.
  • Non-errored P0 traces are still dropped; stats still count everything.
  • Rate limits are keyed per-env, so services don't share one budget.

The sampling logic lives in the dependency-free datadog-agent-trace-sampler crate (DataDog/serverless-components#141): primitives in (SpanView/TraceView), a SampleDecision out, no protobuf Span type, so consumers pinning different libdatadog revisions can share it.

Configuration

Variable Default Notes
DD_APM_ERROR_SAMPLER_ENABLED false Rescue errored traces that automatic sampling dropped.

Flat key (apm_error_sampler_enabled), unlike the Go agent's nested apm_config.*.

The crate offers two strategies; this PR ships AlwaysKeep (rescue every errored auto-dropped trace), since Lambda's per-invocation volume is low and freeze/thaw breaks RateLimited's 30s window. RateLimited (a traces/sec budget à la the Go agent's errors_per_second) stays unwired, along with DD_APM_ERROR_TPS/DD_APM_EXTRA_SAMPLE_RATE.

When disabled, the rescue path — including the clock read — is skipped entirely, so behavior is unchanged from before this PR. When enabled, a failed clock read now falls back to 0 instead of aborting the extension.

Blocked on

DataDog/serverless-components#141: the four pins in bottlecap/Cargo.toml point at its branch rev (54e570a) and need repinning to main once merged.

Testing

Unit tests in traces/trace_processor.rs:

  • test_error_sampler_rescues_errored_p0_chunks — errored auto-dropped chunk is rescued and stamped; non-errored stays dropped; explicit user drop is never rescued.
  • test_error_sampler_rescues_chunk_with_errored_child_span — a healthy root with an errored child is still rescued.
  • test_disabled_error_sampler_drops_errored_p0_chunks — shipping default (disabled) still drops errored P0 traces.

Also: config tests for apm_error_sampler_enabled's default/overrides; apm_integration_test.rs runs the existing APM assertions with the sampler enabled; prod and tests share one constructor (new_error_sampler) so tests exercise the shipping config.

cargo test — 567 passed. cargo clippy --all-targets and cargo fmt --check clean.

🤖

@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Aug 3, 2026

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

🚦 4 Pipeline jobs failed

DataDog/datadog-lambda-extension | e2e-test-status (amd64)   View in Datadog   GitLab

DataDog/datadog-lambda-extension | e2e-test-status (amd64, fips)   View in Datadog   GitLab

DataDog/datadog-lambda-extension | integration-cleanup-layer   View in Datadog   GitLab

View all 4 failed jobs.

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e822993 | Docs | Datadog PR Page | Give us feedback!

@lucaspimentel

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: ffbb0823a7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an agent-side error sampler to the Bottlecap trace processing pipeline so that errored traces that were auto-dropped (P0/AutoDrop) on the lambda_extension_compute_stats path can be “rescued” up to a configurable TPS budget, improving error visibility while still keeping non-errored P0 traces dropped and honoring explicit user drops.

Changes:

  • Introduces an ErrorsSampler into ServerlessTraceProcessor and uses it to selectively retain errored AutoDrop chunks, stamping _dd.errors_sr on rescued roots.
  • Adds config surface area for DD_APM_ERROR_TPS and DD_APM_EXTRA_SAMPLE_RATE (defaults + env/YAML tests).
  • Wires the sampler into the runtime entrypoint and updates unit/integration tests and third-party licensing/deps to include the new shared sampler crate.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
bottlecap/src/traces/trace_processor.rs Implements the rescue decision path for errored AutoDrop chunks and adds unit tests for the new behavior.
bottlecap/src/config/mod.rs Adds apm_error_tps / apm_extra_sample_rate configuration fields and parsing/tests.
bottlecap/src/bin/bottlecap/main.rs Constructs and injects the error sampler into the trace processor using config values.
bottlecap/tests/apm_integration_test.rs Updates the integration pipeline wiring to include the new sampler field.
bottlecap/src/lifecycle/invocation/processor.rs Updates test constructions of ServerlessTraceProcessor to provide an error sampler.
bottlecap/Cargo.toml Adds the datadog-agent-trace-sampler dependency and updates serverless-components rev pins.
bottlecap/Cargo.lock Locks the new sampler crate and updates the serverless-components sources to the new rev.
bottlecap/LICENSE-3rdparty.csv Adds third-party license metadata for datadog-agent-trace-sampler.

Comment thread bottlecap/src/traces/trace_processor.rs Outdated
Comment thread bottlecap/src/traces/trace_processor.rs Outdated
Comment thread bottlecap/src/traces/trace_processor.rs Outdated
@lucaspimentel
lucaspimentel marked this pull request as ready for review August 6, 2026 20:27
@lucaspimentel
lucaspimentel requested review from a team as code owners August 6, 2026 20:27
lucaspimentel and others added 8 commits August 6, 2026 17:56
On the lambda_extension_compute_stats path, the extension drops every
trace marked P0 (priority <= 0) after computing its stats. This adds an
error sampler that gives those dropped traces a second look: errored
traces are kept (rescued) up to DD_APM_ERROR_TPS traces/sec (default 10),
distributed fairly across trace signatures, with _dd.errors_sr stamped on
the rescued root span. Non-errored P0 traces are still dropped, and stats
still count all traces. This guarantees error visibility even under
aggressive sampling.

Ports the Go trace agent's ScoreSampler/ErrorTPS behavior via the shared,
dependency-free datadog-agent-trace-sampler crate.

New config:
- DD_APM_ERROR_TPS (default 10.0; 0 disables the rescue)
- DD_APM_EXTRA_SAMPLE_RATE (default 1.0)

🤖
Errored traces were only rescued from a drop decision when the root span
itself carried the error, so a trace whose failure happened deeper (for
example a failed downstream call that the handler caught) was still
dropped. Now an error anywhere in the trace makes it a rescue candidate,
matching the Datadog Agent.

🤖
Traces dropped on purpose, either by a tracer sampling rule or by an
explicit MANUAL_DROP, were being fed to the error sampler and could be
sent to Datadog anyway when they contained an error. Only traces dropped
by automatic sampling are now rescue candidates, matching the Datadog
Agent.

🤖
The error sampler's per-signature rate limits were keyed on the
extension's own DD_ENV, so when that was unset every trace shared one
empty env and distinct services competed for the same budget. The env
the tracer reported with the trace is now used instead, matching the
Datadog Agent.

🤖
A panic while the error sampler's lock was held poisoned the mutex,
which silently disabled error-trace rescue for the rest of the sandbox's
life. Recover through poisoning instead: the sampler holds only
rolling-window counters, so a partially updated bucket costs far less
than losing the feature entirely.

🤖
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
The disabled check re-derived "sampler is off" from apm_error_tps at the
call site, a second definition of a condition the sampler already knows.
It now asks the sampler via the new is_disabled(), hoisted above the
payload loop so it costs one lock per flush instead of a check per chunk.

Repins the four serverless-components deps to pick up is_disabled(),
which also brings in non-finite client sample rate handling.

🤖
@lucaspimentel
lucaspimentel force-pushed the lpimentel/add-trace-error-sampler branch from 7a0f64f to 328cf05 Compare August 6, 2026 21:57
Bump the serverless-components pin to 54e570ae, which adds the
dual-mode ErrorSamplerMode (AlwaysKeep | RateLimited) to
datadog-agent-trace-sampler and makes `mode` a required field on
ErrorSamplerConfig. The rev also carries a fix for non-finite client
sample rates on the error sample rate path.

Hardcode the production sampler to AlwaysKeep: Lambda's per-invocation
trace volume is low, so the RateLimited budget rarely binds, and
freeze/thaw breaks its 30s wall-clock window. Wiring the mode through
config is deferred to a follow-up.

Correct the doc comments on apm_error_tps, apm_extra_sample_rate, and
ServerlessTraceProcessor::error_sampler, which described RateLimited
behavior that no longer runs in production.

Refs APMSVLS-469
Replace apm_error_tps and apm_extra_sample_rate with a single
apm_error_sampler_enabled toggle (DD_APM_ERROR_SAMPLER_ENABLED).

Both replaced knobs were introduced earlier on this branch and never
released, so there is no compatibility constraint. Neither carries its
advertised meaning in AlwaysKeep mode: extra_sample_rate is ignored
outright, and error_tps degrades to an on/off switch, so
DD_APM_ERROR_TPS=25 and =1 behave identically. Exposing a rate cap that
is not enforced is worse than not exposing one. Both return, with their
real semantics, when RateLimited is wired up.

Default false while the feature rolls out as opt-in; the plan is to flip
it once it has soaked. AlwaysKeep derives its disabled flag from
target_tps <= 0.0, so the boolean maps onto 1.0 / 0.0 and the disabled
path short-circuits before any SpanView is built.

Refs APMSVLS-469
Build the shipped error sampler (AlwaysKeep, enabled by
apm_error_sampler_enabled) in one place so tests exercise the same
configuration that ships, and cover the disabled default with a test.

A failed clock read no longer aborts the extension, and the clock is only
read when error rescue is enabled. In AlwaysKeep mode the sampler ignores
span contents, so only the root span view is built instead of one per
span.

🤖
The float_cmp allow no longer matches any comparison in the test module
and would mask a real one.

🤖
Condense the doc and inline comments added with the error sampler: drop
roadmap notes, references to prior behavior, and comments that restate
the code they sit above.

🤖
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants